Layered and distributed grid-specific network services

ABSTRACT

In one embodiment, a layered/distributed grid-specific network services system comprises grid sensors in the utility grid configured to generate grid data values such as raw grid data values, processed grid data values, and/or any combination thereof, and to communicate the grid data values using a communication network. Distributed grid devices in the utility grid may be configured to receive the grid data values, and one or more of the grid devices may be configured to convert raw grid data values into processed grid data values. Application devices in the utility grid may be configured to access the grid data values from the distributed grid devices, and to further process the grid data values according to a particular grid application operating at the corresponding application device into application data values.

RELATED APPLICATIONS

The present application claims priority to U.S. Provisional ApplicationNo. 61/491,377, filed May 31, 2011, entitled VARIABLE TOPOLOGYDISTRIBUTED INTELLIGENCE FOR SMART GRIDS, by Jeffrey D. Taft, thecontents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to utility control systems,e.g., to “smart grid” technologies.

BACKGROUND

Utility control systems and data processing systems have largely beencentralized in nature. Energy Management Systems (EMS's), DistributionManagement Systems (DMS's), and Supervisory Control and Data Acquisition(SCADA) systems reside in control or operations centers and rely uponwhat have generally been low complexity communications to field devicesand systems. There are a few distributed control systems for utilityapplications, including a wireless mesh system for performing faultisolation using peer-to-peer communications among devices on feedercircuits outside of the substations. In addition, certain protectionschemes involve substation-to-substation communication and localprocessing. In general however, centralized systems are the primarycontrol architecture for electric grids.

Moreover, conventional grid functions and applications are typically“siloed,” meaning that each application is independent and specific toits task; however, many of these siloed applications/functions haveoverlapping functionality. For example, a first application such asvolt/VAr regulation may require voltage readouts, and a secondapplication such as a outage detection may also require the voltagereadouts. Currently, each of the siloed applications is required toindependently obtain the voltage readouts.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments herein may be better understood by referring to thefollowing description in conjunction with the accompanying drawings inwhich like reference numerals indicate identically or functionallysimilar elements, of which:

FIG. 1 illustrates an example simplified utility grid hierarchy;

FIG. 2 illustrates an example simplified communication network based ona utility grid (e.g., a “smart grid” network);

FIG. 3 illustrates an example simplified device/node;

FIG. 4 illustrates an example table showing challenges associated withcomplexity for smart grids at scale;

FIG. 5 illustrates an example of a smart grid core functions stack;

FIG. 6 illustrates an example of various feedback arrangements;

FIG. 7 illustrates an example chart showing a latency hierarchy;

FIG. 8 illustrates an example table of data lifespan classes;

FIG. 9 illustrates an example of an analytics architecture;

FIG. 10 illustrates an example of types of distributed analyticelements;

FIG. 11 illustrates an example data store architecture;

FIGS. 12A-12E illustrate an example layered services architecture model(“stack”);

FIG. 13 illustrates an example logical stack for a distributedintelligence platform;

FIGS. 14A-14D illustrate an example of a layered services platform;

FIG. 15 illustrates another example of a smart grid function stack;

FIG. 16 illustrates an example of a generalized structure of alayered/distributed grid-specific network services system;

FIG. 17 illustrates another example of a layered/distributedgrid-specific network services system correlated with a smart gridfunction stack;

FIG. 18 illustrates an example of hierarchical ordered data generationas applied to a raw voltage data value; and

FIG. 19 illustrates an example simplified procedure for alayered/distributed grid-specific network services system.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

According to one or more embodiments of the disclosure, a system thatprovides layered/distributed grid-specific network services comprises aplurality of grid sensors, a plurality of distributed grid devices, anda plurality of application devices in a utility grid. The plurality ofgrid sensors in the utility grid may be configured to generate grid datavalues such as, for example, raw grid data values, processed grid datavalues, and/or any combination thereof, and to communicate the grid datavalues using a communication network. The plurality of distributed griddevices in the utility grid may be configured to receive the grid datavalues, and one or more of the plurality of grid devices may beconfigured to convert raw grid data values into processed grid datavalues. The plurality of application devices in the utility grid may beconfigured to access the grid data values from the distributed griddevices, and to further process the grid data values according to aparticular grid application operating at the corresponding applicationdevice into application data values.

Description

Electric power is generally transmitted from generation plants to endusers (industries, corporations, homeowners, etc.) via a transmissionand distribution grid consisting of a network of interconnected powerstations, transmission circuits, distribution circuits, and substations.Once at the end users, electricity can be used to power any number ofdevices. Generally, various capabilities are needed to operate powergrids at the transmission and distribution levels, such as protection,control (flow control, regulation, stabilization, synchronization),usage metering, asset monitoring and optimization, system performanceand management, etc.

FIG. 1 illustrates an example simplified utility grid and an examplephysical hierarchy of electric power distribution. In particular, energymay be generated at one or more generation facilities 110 (e.g., coalplants, nuclear plants, hydro-electric plants, wind farms, etc.) andtransmitted to one or more transmission substations 120. From thetransmission substations 120, the energy is next propagated todistribution substations 130 to be distributed to various feedercircuits (e.g., transformers) 140. The feeders 140 may thus “feed” avariety of end-point “sites” 150, such as homes, buildings, factories,etc. over corresponding power-lines.

Note that the illustrative structure of the utility grid is shown as ahighly simplified hierarchy, e.g., a hierarchy with generation at thetop, transmission substations as the next tier, distribution substationas the next, etc. However, those skilled in the art will appreciate thatFIG. 1 is merely an illustration for the sake of discussion, and actualutility grids may operate in a vastly more complicated manner (e.g.,even in a vertically integrated utility). That is, FIG. 1 illustrates anexample of power-based hierarchy (i.e., power starts at the generationlevel, and eventually reaches the end-sites), and not a logicalcontrol-based hierarchy. In particular, in conventional environments,transmission and primary distribution substations are at the samelogical level, while generation is often its own tier and is reallycontrolled via automatic generation control (AGC) by a BalancingAuthority or other Qualified Scheduling Entity, whereas transmissionlines and substations are under the control of a transmission operatorEnergy Management System (EMS). Primary distribution substations may becontrolled by a transmission EMS in some cases and are controlled by adistribution control center, such as when distribution is via aDistribution System Operator (DSO). (Generally, distribution feeders dologically belong to primary distribution substations as shown.)

In the case of distributed control, that is, in terms of control-basedhierarchy, substations may be grouped so that some are logically higherlevel than others. In this manner, the need to put fully duplicatedcapabilities into each substation may be avoided by allocatingcapabilities so as to impose a logical control hierarchy onto anotherwise flat architecture, such as according to the techniquesdescribed herein. In such cases, transmission substations may be groupedand layered, while primary distribution substations may be separatelygrouped and layered, but notably it is not necessary (or even possible)that distribution substations be logically grouped under transmissionsubstations.

In general, utility companies can benefit from having accuratedistribution feeder (medium voltage/low voltage or “MV/LV” circuit)connectivity information in their software applications and data stores.This is especially useful for outage management and for convenientapplication to planning, construction, operations, and maintenance. Itis, however, very challenging to try to construct or approximate thecircuit model within a geographic information systems (GIS) environmentdue to the complexity of modeling the dynamic nature of an electricalnetwork. That is, while the utility may have an “as-built” database, itmay differ from the actual grid for various reasons, includinginaccurate or incomplete data capture on grid construction, changes tocircuits that are not reflected in updates to the database, andstructural damage to the grid. In addition, circuit topology may changedynamically as feeder switches are operated in the course of eithernormal or emergency operations. Such changes result in an “as-operated”topology that is dynamic and is not reflected in the “as-built”database.

To assist in control of the utility grid, various measurement andcontrol devices may be used at different locations within the grid 100.Such devices may comprise various energy-directing devices, such asreclosers, power switches, circuit breakers, etc. In addition, othertypes of devices, such as sensors (voltage sensors, current sensors,temperature sensors, etc.) or computational devices, may also be used.Electric utilities use alternating-current (AC) power systemsextensively in generation, transmission, and distribution. Most of thesystems and devices at the high and medium voltage levels operate onthree-phase power, where voltages and currents are grouped in threes,with the waveforms staggered evenly. The basic mathematical object thatdescribes an AC power system waveform (current of voltage) is the“phasor” (phase angle vector). Computational devices known as PhasorMeasurement Units (PMUs) have thus been commercialized by severalcompanies to calculate phasors from power waveforms. Because phase angleis a relative quantity, it is necessary when combining phasors takenfrom different parts of a power grid to align the phase angle elementsto a common phase reference; this has been typically done in PMUsthrough the use of GPS timing signals. Such phasors are known assynchrophasors.

FIG. 2 is a schematic block diagram of a communication network 200 thatmay illustratively be considered as an example utility gridcommunication network. The network 200 illustratively comprisesnodes/devices interconnected by various methods of communication, suchas wired links or shared media (e.g., wireless links, Power-linecommunication (PLC) links, etc.), where certain devices, such as, e.g.,routers, sensors, computers, etc., may be in communication with otherdevices, e.g., based on distance, signal strength, current operationalstatus, location, etc. Those skilled in the art will understand that anynumber of nodes, devices, links, etc. may be used in the computernetwork, and that the view shown herein is for simplicity. Data packetsmay be exchanged among the nodes/devices of the computer network 200using predefined network communication protocols such as certain knownwired protocols, wireless protocols (e.g., IEEE Std. 802.15.4, WiFi,Bluetooth®, DNP3 (distributed network protocol), Modbus, IEC 61850,etc.), PLC protocols, or other protocols where appropriate. In thiscontext, a protocol consists of a set of rules defining how the nodesinteract with each other.

Illustratively, a control center 210 (and backup control center 210 a)may comprise various control system processes 215 and databases 217interconnected via a network switch 219 to a system control network 205.Additionally, one or more substations 220 may be connected to thecontrol network 205 via switches 229, and may support variousservices/process, such as a distributed data service 222, grid stateservice (e.g., “parstate”, a determination of part of the whole gridstate) 223, control applications 225, etc. The substations 220 may alsohave a GPS clock 221 to provide timing, which may be distributed to theFARs 250 (below) using IEEE Std. 1588. Note that a monitoring center 230may also be in communication with the network 205 via a switch 239, andmay comprise various analytics systems 235 and databases 237. Thesubstations 220 may communicate with various other substations (e.g.,from transmission substations to distribution substations, as mentionedabove) through various methods of communication. For instance, ahierarchy of wireless LAN controllers (WLCs) 240 and field area routers(FARs) 250 may provide for specific locality-based communication betweenvarious portions of the underlying utility grid 100 in FIG. 1. WLCs 240(which may also be considered as a type of higher grid level FAR) maycomprise various services, such as data collection 245, controlapplications 246, etc. Generally, grid devices on shared feeder sections(e.g., FAR 250-X) may communicate with both involved substations (e.g.,both WLCs 240, as shown). Further, FARs 250 may also comprise datacollection services 255 themselves, and may collect data from (ordistribute data to) one or more end-point communication devices 260,such as sensors and/or actuators (e.g., home energy controllers, gridcontrollers, etc.).

Specific details of the operation of the smart grid devices aredescribed below. Note that while there is a general correlation betweenthe communication network 200 and underlying utility grid 100 (e.g.,control centers, substations, end-points, etc.), such a correlation mayonly be generally assumed, and is not a necessity. For instance, FARs250 may be associated with feeder circuits 140, or may be more granularsuch as, e.g., “pole-top” routers. In other words, the hierarchies shownin FIG. 1 and FIG. 2 are not meant to be specifically correlated, andare merely examples of hierarchies for illustration.

FIG. 3 is a schematic block diagram of an example node/device 300 thatmay be used with one or more embodiments described herein, e.g., as anycapable “smart grid” node shown in FIG. 2 above. In particular, thedevice 300 is a generic and simplified device, and may comprise one ormore network interfaces 310 (e.g., wired, wireless, PLC, etc.), at leastone processor 320, and a memory 340 interconnected by a system bus 350,as well as a power supply 360 (e.g., battery, plug-in, etc.).

The network interface(s) 310 contain the mechanical, electrical, andsignaling circuitry for communicating data over links coupled to thenetwork 200. The network interfaces may be configured to transmit and/orreceive data using a variety of different communication protocols. Note,further, that the nodes may have two different types of networkconnections 310, e.g., wireless and wired/physical connections, and thatthe view herein is merely for illustration. Also, while the networkinterface 310 is shown separately from power supply 360, for PLC thenetwork interface 310 may communicate through the power supply 360, ormay be an integral component of the power supply. In some specificconfigurations the PLC signal may be coupled to the power line feedinginto the power supply.

The memory 340 of the generic device 300 comprises a plurality ofstorage locations that are addressable by the processor 320 and thenetwork interfaces 310 for storing software programs and data structuresassociated with the embodiments described herein. Note that certaindevices may have limited memory or no memory (e.g., no memory forstorage other than for programs/processes operating on the device andassociated caches). The processor 320 may comprise necessary elements orlogic adapted to execute the software programs and manipulate the datastructures 345. An operating system 342, portions of which are typicallyresident in memory 340 and executed by the processor, functionallyorganizes the device by, inter alia, invoking operations in support ofsoftware processes and/or services executing on the device. Thesesoftware processes and/or services may comprise one or moregrid-specific application processes 348, as described herein. Note thatwhile the grid-specific application process 348 is shown in centralizedmemory 340, alternative embodiments provide for the process to bespecifically operated within the network elements or network-integratedcomputing elements 310.

It will be apparent to those skilled in the art that other processor andmemory types, including various computer-readable media, may be used tostore and execute program instructions pertaining to the techniquesdescribed herein. Also, while the description illustrates variousprocesses, it is expressly contemplated that various processes may beembodied as modules configured to operate in accordance with thetechniques herein (e.g., according to the functionality of a similarprocess). Further, while the processes have been shown separately, thoseskilled in the art will appreciate that processes may be routines ormodules within other processes.

As noted above, utility control systems and data processing systems havelargely been centralized in nature. Energy Management Systems (EMS's),Distribution Management Systems (DMS's), and Supervisory Control andData Acquisition (SCADA) systems reside in control or operations centersand rely upon what have generally been low complexity communications tofield devices and systems. Both utilities and makers of various gridcontrol systems have recognized the value of distributed intelligence,especially at the distribution level.

Generally, distributed intelligence is defined as the embedding ofdigital processing and communications ability in a physically dispersed,multi-element environment (specifically the power grid infrastructure,but also physical networks in general). In the area of sensing,measurement and data acquisition, key issues are:

-   -   Sensing and measurement—determination of quantities to be        sensed, type and location of sensors, and resulting signal        characteristics;    -   Data acquisition—collection of sensor data, sensor data        transport;    -   System state and observability—key concepts that can be used to        guide the design of sensor systems for physical systems with        topological structure and system dynamics; and    -   Sensor network architecture—elements, structure, and external        properties of sensor networks.        Key elements of distributed intelligence comprise:    -   Distributed data collection and persistence—measurement of        electrical grid state, power quality, asset stress and        utilization factors, environmental data, real-time grid        topology, and device operating states, as opposed to central        SCADA;    -   Distributed data transformation and analytics—processing of        measured data and event messages generated by smart grid devices        and systems to extract useful information, prepare data for use        by applications, or to correlate and filter data and events for        aggregation purposes, as opposed to data center processing; and    -   Distributed control—execution of actual control algorithms, with        control commands being sent directly to grid control actuators        for relatively local controllers, as opposed to central control.

By establishing the network as a platform (NaaP) to support distributedapplications, and understanding the key issues around sensing andmeasurement for dynamic physical network systems, key capabilities ofsmart communication networks may be defined (e.g., as described below)that support current and future grid applications. In particular, as ICT(Information Communication Technology) networks converge with physicalpower grids and as “smart” functions penetrate the grid, centralizedarchitectures for measurement and control become increasinglyinadequate. Distribution of intelligence beyond the control center tolocations in the power grid provides the opportunity to improveperformance and increase robustness of the data management and controlsystems by addressing the need for low latency data paths and supportingvarious features, such as data aggregation and control federation anddisaggregation.

In particular, there are a number of compelling arguments for usingdistributed intelligence in smart power grids, and in large scalesystems in general, such as:

-   -   Low Latency Response—A distributed intelligence architecture can        provide the ability to process data and provide it to the end        device without a round trip back to a control center;    -   Low Sample Time Skew—Multiple data collection agents can easily        minimize first-to-last sample time skew for better system state        snapshots;    -   Scalability—No single choke point for data acquisition or        processing; analytics at the lower levels of a hierarchical        distributed system can be processed and passed on to higher        levels in the hierarchy. Such an arrangement can keep the data        volumes at each level roughly constant by transforming large        volumes of low level data into smaller volumes of data        containing the relevant information. This also helps with        managing the bursty asynchronous event message data that smart        grids can generate (example: last gasp messages from meters        during a feeder momentary outage or sag). The scalability issue        is not simply one of communication bottlenecking however—it is        also (and perhaps more importantly) an issue of data persistence        management, and a matter of processing capacity. Systems that        use a central SCADA for data collection become both memory-bound        and CPU-bound in a full scale smart grid environment, as do        other data collection engines; and    -   Robustness—Local autonomous operation, continued operation in        the presence of fragmentation of the network, graceful system        performance and functional degradation in the face of failures,        etc.

Standard approaches to distributed processing suffer from shortcomingsrelative to the electric grid environment. These shortcomings includeinability to handle incremental rollout, variable distribution ofintelligence, and applications not designed for a distributed (orscalable) environment. Further, existing approaches do not reflect thestructure inherent in power grids and do not provide integration acrossthe entire set of places in the grid where intelligence is located, oracross heterogeneous computing platforms. Current systems also sufferfrom inability to work with legacy software, thus requiring massivesoftware development efforts at the application level to makeapplications fit the platform, and also lack zero-touch deploymentcapability and requisite security measures.

For instance, one major obstacle in the adoption of distributedintelligence, now that IP communications and embedded processingcapabilities are becoming available in forms that utilities can use, isthat utilities cannot make large equipment and system changes in largediscrete steps. Rather they must go through transitions that can takeyears to complete. This is due to the nature of their mission and thefinancial realities utilities must deal with. In practice, utilitiesmust be able to transition from centralized to distributed intelligence,and must be able to operate in a complicated hybrid mode for longperiods of time, perhaps permanently. This means that the utility mustbe able to roll out distributed intelligence incrementally whilemaintain full operations over the entire service area, and must be ableto modify the distributed architecture appropriately over time andgeography. Simply having a distributed architecture implementation isnot sufficient; it must be easily and continually mutable in terms ofwhat functionality is distributed to which processing locations in thegrid and must be capable of coexisting with legacy control systems wherethey remain in place. Therefore, there exist various kinds of variabletopology for effective distributed intelligence:

-   -   Transition Variability—Rollout of distributed intelligence        functions will be uneven both geographically (topologically) and        over time, and there is no one-size-fits-all solution, even for        a single utility;    -   End State Variability—Not every distributed intelligence        function will be pushed to every end node of the same class, and        distributed intelligence functions and distributions will have        to change over the life of the system;    -   Operational Variability—Users must be able to change locations        of functions to deal with failures and maintenance, etc.

Additionally, design and implementation of smart grids at scale poses anumber of challenging architecture issues. Many of these issues are notapparent or do not show significant effects at pilot scale, but canbecome crucial at full scale. Note that generally herein, “at fullscale” means one or more of:

-   -   Endpoint scale—the number of intelligent endpoints is in the        millions per distribution grid;    -   Functional complexity scale—the number and type of functions or        applications that exhibit hidden layer coupling through the grid        is three or more; or the number of control systems (excluding        protection relays) acting on the same feeder section or        transmission line is three or more; and    -   Geospatial complexity—the geographical/geospatial complexity of        the smart grid infrastructure passes beyond a handful of        substation service areas or a simple metro area deployment to        large area deployments, perhaps with interpenetrated service        areas for different utilities, or infrastructure that cuts        across or is shared across multiple utilities and related        organizations.

In the table 400 shown in FIG. 4, some of the challenges arising fromthese levels of complexity for smart grids at scale are illustrated. Forinstance, ultra large scale (ULS) characteristics of smart grids atscale are usually associated with decentralized control, inherentlyconflicting diverse requirements, continuous evolution and deployment,heterogeneous, inconsistent, and changing elements, and various normalfailure conditions. Also, hidden couplings via the grid exist, sincesystems and controls are inherently coupled through grid electricalphysics and therefore interact in ways that may is be unaccounted for insystem designs. The grid may further be viewed at scale as amulti-objective, multi-control system, where multiple controls affectingthe same grid portions, and where some of the controls actually lieoutside of the utility and/or are operating on multiple time scales.Moreover, bulk or aggregate control commands, especially as regardssecondary load control and stabilization, may not consider the specificlocalities within the grid, and are not broken down to the feeder oreven section level, taking into account grid state at the level. Lastly,smart grid-generated data must be used on any of a number of latencyscales, some of which are quite short, thus precluding purelycentralized processing and control approaches. Note that there areadditional issues affecting architecture for smart grids at scale thanthose that are shown in FIG. 4, but these are representative of some ofthe key challenges.

The smart grid has certain key attributes that lead to the concept ofcore function classes supported by the smart grid. These key attributesinclude:

-   -   A geographically distributed analog infrastructure;    -   A digital superstructure consisting of digital processing        layered on top of the analog superstructure, along with        ubiquitous IP-based digital connectivity; and    -   Embedded processors and more general smart devices connected to        the edges of the smart grid digital superstructure and the        analog infrastructure; these include both measurement (sensor)        and control (actuator) devices.

Given this environment, and given our present understanding of thenature of the desired behavior of the power grid, we may identify anumber of key function classes; functional groups that arise inherentlyfrom the combination of desired smart grid behavior, grid structure, andthe nature of the digital superstructure applied to the grid. Anunderstanding of these core function groups is key to developing a viewtoward a layered network services architecture for smart grids. A modelis presented herein in which smart grid applications of any type arebuilt upon a logical platform of core function classes that arise fromthe grid itself.

FIG. 5 illustrates the concept and the function classes themselves. Forinstance, to support distributed intelligence for electric power gridsor any physical network, the concept of network services may be extendedto become a stack of service groups, where the services becomeincreasingly domain-oriented as one moves up the stack. This means thatthe lower layer contains ordinary network services. The next layercontains services that support distributed intelligence. The third layerprovides services that support domain specific core functions. The toplayer provides services that support application integration forreal-time systems.

Specifically, as shown in the model of FIG. 5, the function classes aredivided into four tiers.

1) The base tier 510 is:

-   -   Power Delivery Chain Unification: use of digital communications        to manage secure data flows and to integrate virtualized        information services at low latency throughout the smart grid;        enable N-way (not just two-way) flow of smart grid information;        provision of integration through advanced networking protocols,        converged networking, and service insertion. Note that this        layer is based on advanced networking and communication, and in        general may be thought of as system unification. In this model,        networking plays a foundational role; this is a direct        consequence of the distributed nature of smart grid assets.

2) The second tier 520 is:

-   -   Automatic Low Level Control 521—digital protection inside and        outside the substation, remote sectionalizing and automatic        reclosure, feeder level flow control, local automatic        voltage/VAr regulation, stabilization, and synchronization; and    -   Remote Measurement 522—monitoring and measurement of grid        parameters and physical variables, including direct power        variables, derived element such as power quality measures, usage        (metering), asset condition, as-operated topology, and all data        necessary to support higher level function classes and        applications.

3) The third tier 530 is:

-   -   Control Disaggregation 531—control commands that are calculated        at high levels must be broken down into multiple commands that        align with the conditions and requirements at each level in the        power delivery chain; the process to accomplish this is the        logical inverse of data aggregation moving up the power delivery        chain, and must use knowledge of grid topology and grid        conditions to accomplish the disaggregation; and    -   Grid State Determination 532—electrical measurement, power state        estimation, and visualization, voltage and current phasors, bus        and generator phase angles, stability margin, real and reactive        power flows, grid device positions/conditions, DR/DSM available        capacity and actual response measurement, storage device charge        levels, circuit connectivity and device parametrics.

4) The fourth tier 540 is:

-   -   Fault Intelligence 541—detection of short or open circuits and        device failures; fault and failure classification,        characterization (fault parameters), fault location        determination, support for outage intelligence, support for        adaptive protection and fault isolation, fault prediction, fault        information notification and logging;    -   Operational Intelligence 542—all aspects of information related        to grid operations, including system performance and operational        effectiveness, as well as states of processes such as outage        management or fault isolation;    -   Outage Intelligence 543—detection of service point loss of        voltage, inside/outside trouble determination, filtering and        logging of momentaries, extent mapping and outage verification,        root cause determination, restoration is tracking and        verification, nested root cause discovery, outage state and        process visualization, crew dispatch support;    -   Asset Intelligence 544—this has two parts:        -   asset utilization intelligence—asset loading vs. rating,            peak load measurement (amplitude, frequency), actual demand            curve measurement, load/power flow balance measurement,            dynamic (real-time) de-rating/re-rating, real-time asset            profitability/loss calculation; and        -   asset health/accumulated stress intelligence—device health            condition determination, online device and system failure            diagnostics, device failure or imminent failure            notification, asset accumulated stress measurement, Loss of            Life (LoL) calculation, Estimated Time to Failure (ETTF)            prediction, Asset Failure System Risk (AFSR) calculation;            and    -   Control Federation 545—grid control increasingly involves        multiple control objectives, possible implemented via separate        control systems. It is evolving into a multi-controller,        multi-objective system where many of the control systems want to        operate the same actuators. A core function of the smart grid is        to federate these control systems that include Demand Response        and DSM, voltage regulation, capacitor control, power flow        control, Conservation Voltage Reduction (CVR), Electric Vehicle        Charging Control, Line Loss Control, Load Balance Control,        DSTATCOM and DER inverter VAr control, reliability event        control, Virtual Power Plant (VPP) control, and meter        connect/disconnect and usage restriction control.

These function classes may support one or more smart grid applications550. In general, therefore, smart grid networks, that is, thecombination of a utility grid with a communication network, along withdistributed intelligent devices, may thus consist of various type ofcontrol, data acquisition (e.g., sensing and measurement), anddistributed analytics, and may be interconnected through a system ofdistributed data persistence. Examples may include, among others,distributed SCADA data collection and aggregation, grid statedetermination and promulgation, implementation of distributed analyticson grid data, control command delivery and operational verification,control function federation (merging of multiple objective/multiplecontrol systems so that common control elements are used innon-conflicting ways), processing of events streams from grid devices tofilter, prevent flooding, and to detect and classify events for lowlatency responses, and providing virtualization of legacy grid devicesso that they are compatible with modern approaches to device operationand network security.

In particular, there may be a number of types of control, such assequence control (e.g., both stateless and stateful, typified byswitching systems of various kinds), stabilizers (e.g., which moderatedynamic system behavior, typically through output or state feedback sothat the system tends to return to equilibrium after a disturbance), andregulators (e.g., in which a system is made to follow the dynamics of areference input, which may be dynamic or static set points). Quiteoften, all three of these are present in the same control system. Interms of electric power grids, flow control is sequence control, whereasmodel power oscillation damping and volt/VAr control representstabilization and regulatory control, respectively.

For most control systems, feedback is a crucial component. FIG. 6illustrates output feedback 610 and state feedback 620, both of whichare quite common. FIG. 6 also illustrates a slightly more complexfeedback arrangement 630 intended to be used when a system exhibits twovery different sets of dynamics, one fast and one slow. There are agreat many extensions of the basic control loop and the volume ofmathematics, theory, and practice is enormous and widely used.

Regarding data acquisition, sensing and measurement support multiplepurposes in the smart grid environment, which applies equally as well tomany other systems characterized by either geographic dispersal, orlarge numbers of ends points, especially when some form of control isrequired. Consequently, the sensing system design can be quite complex,involving issues physical parameter selection, sensor mix and placementoptimization, measurement type and sample rate, data conversion, sensorcalibration, and compensation for non-ideal sensor characteristics.

Additionally, collection of the data in large scale systems such assmart grids presents issues of cycle time, data bursting, and sampleskew. There are multiple modes of data collection for large scalesystems and each presents complexities, especially when the system modelinvolves transporting the data to a central location. In the typicalround-robin scanning approach taken by many standard SCADA systems, thetime skew between first and last samples represents an issue for controlsystems that is insignificant when the scan cycle time is short comparedto system dynamics, but as dynamics increase in bandwidth with advancedregulation and stabilization, and as the number of sensing pointsincreases, the sample time skew problem becomes significant.

Data is consumed in a variety of ways and places in a power grid; mostof these are not located at the enterprise data center and much griddata does not enter the data center. Some of it does not even enter thecontrol/operations center, as it must be consumed “on the fly” in griddevices and systems. Consequently it is important to classify dataaccording to the latency requirements of the devices, systems, orapplications that use it and appropriate persistence (or lack thereof)must also be defined. Note that much grid data has multiple uses; infact, it is an element of synergy that has significant impact on smartgrid economics and system design (networking, data architecture,analytics) to ensure that data is used to support as many outcomes aspossible.

FIG. 7 is a chart 700 that illustrates the issue of latency, as latencyhierarchy is a key concept in the design of both data management andanalytics applications for physical networks with control systems orother real-time applications. In particular, in the example (andnon-limiting) chart 700, grid sensors and devices are associated with avery low latency, where high-speed/low-latency real-time analytics mayrequire millisecond to sub-second latency to provide results through amachine-to-machine (M2M) interface for various protection and controlsystems. The latency hierarchy continues toward higher latencyassociations as shown and described in chart 700, until reaching a veryhigh latency at the business data repository level, where data withindays is to months may be used for business intelligence processing, andtransmitted via a human-machine interface (HMI) for various reporting,dashboards, key performance indicators (KPI's), etc. Note that the chartdoes not illustrate that a given data element may in fact have multiplelatency requirements, depending on the various ways it may be used,meaning that any particular datum may have multiple destinations.

The latency hierarchy issue is directly connected to the issue oflifespan classes, meaning that depending on how the data is to be used,there are various classes of storage that may have to be applied. Thistypically results in hierarchical data storage architecture, withdifferent types of storage being applied at different points in the gridthat correspond to the data sources and sinks, coupled with latencyrequirements.

FIG. 8 illustrates a table 800 listing some types of data lifespanclasses that are relevant to smart grid devices and systems. Inparticular, transit data exists for only the time necessary to travelfrom source to sink and be used; it persists only momentarily in thenetwork and the data sink and is then discarded. Examples are an eventmessage used by protection relays, and sensor data used in closed loopcontrols; persistence time may be microseconds. On the other hand,burst/flow data, which is data that is produced or processed in bursts,may exist temporarily in FIFO (first in first out) queues or circularbuffers until it is consumed or overwritten. Examples of burst/flow datainclude telemetry data and asynchronous event messages (assuming theyare not logged), and often the storage for these types of data areincorporated directly into applications, e.g., CEP engine event buffers.Operational data comprises data that may be used from moment to momentbut is continually updated with refreshed values so that old values areoverwritten since only present (fresh) values are needed. Examples ofoperational data comprise grid (power) state data such as SCADA datathat may be updated every few seconds. Transactional data exists for anextended but not indefinite time, and is typically used in transactionprocessing and business intelligence applications. Storage oftransactional data may be in databases incorporated into applications orin data warehouses, datamarts or business data repositories. Lastly,archival data is data that must be saved for very long (even indefinite)time periods, and typically includes meter usage data (e.g., sevenyears), PMU data at ISO/RTO's (several years), log files, etc. Note thatsome data may be retained in multiple copies; for example, ISO's mustretain PMU data in quadruplicate. Just as with latency hierarchy, griddata may progress through various lifetime classes as it is used indifferent ways. This implies that some data will migrate from one typeof data storage to another as its lifetime class changes, based on howit is used.

Distributed analytics may be implemented in a fully centralized manner,such as usually done with Business Intelligence tools, which operate ona very large business data repository. However, for real-time systems, amore distributed approach may be useful in avoiding the inevitablebottlenecking. A tool that is particularly suited to processing twoclasses of smart grid data (streaming telemetry and asynchronous eventmessages) is Complex Event Processing (CEP) which has lately also beencalled streaming database processing. CEP and its single streampredecessor Event Stream Processing (ESP) can be arranged into ahierarchical distributed processing architecture that efficientlyreduces data volumes while preserving essential information embodies inmultiple data streams.

FIG. 9 shows an example of such analytics architecture. In this case,the analytics process line sensor data and meter events for fault andoutage intelligence. In particular, various line sensors 905 maytransmit their data via ESPs 910, and may be collected by a feeder CEP915 at a substation 920. Substation CEPs 925 aggregate the feeder CEPdata, as well as any data from substation devices 930, and this data maybe relayed to a control center CEP 935 within a control center 940.Along with meter events from meter DCE 945 and other data from database950, the control center CEP 935 may thus perform a higher level ofanalytics than any of the below levels of CEPs, accordingly.

In general, distributed analytics can be decomposed into a limited setof analytic computing elements (“DA” elements), with logical connectionsto other such elements. Full distributed analytics can be constructed bycomposing or interconnecting basic analytic elements as needed. Fivebasic types of distributed analytic elements are defined herein, andillustrated in FIG. 10:

-   -   1. Local loop 1010—an analytic element operates on data reports        its final result to is a consuming application such as a low        latency control;    -   2. Upload 1020—an analytic element operates on data and then        reports out its final result;    -   3. Hierarchical 1030—two or more analytic elements operate on        data to produce partial analytics results which are then fused        by a higher level analytics element, which reports the result;    -   4. Peer to peer 1040—two or more analytics elements operate on        data to create partial results; they then exchange partial        results to compute final result and each one reports its unique        final analytic; and    -   5. Database access 1050—an analytic element retrieves data from        a data store in addition to local data; it operates on both to        produce a result which can be stored in the data store or        reported to an application or another analytic element

A sixth type, “generic DA node” 1060, may thus be constructed torepresent each of the five basic types above.

Given the above-described concept of distributed analytics, includingthe database access element 1050 shown in FIG. 10, it becomes useful toconsider distributed data persistence as an architectural element. Lowlevel and low latency analytics for smart grids (mostly related tocontrol) require state information and while local state components aregenerally always needed, it is often the case that elements of globalstate are also necessary. Operational data (essentially extended systemstate) may be persisted in a distributed operational data store. Thereason for considering a true distributed data store is for scalabilityand robustness in the face of potential network fragmentation. In powersystems, it is already common practice to implement distributed timeseries (historian) databases at the control center and primarysubstation levels. The techniques described herein may incorporate thisand the distributed operational data store into an integrated dataarchitecture by employing data federation in conjunction with variousdata stores.

FIG. 11 illustrates a data store architecture 1100 that federatesdistributed and centralized elements in order to support a wide range ofanalytics, controls, and decision support for business processes. Inparticular, a control center 1110 may comprise various centralizedrepositories or databases, such as a waveform repository 1112, anoperational (Ops) data database 1114, and a time series database 1116.For instance, common interface model (CIM) services 1118 within thecontrol center 1110 may operate based on such underlying data, as may beappreciated in the art. The data itself may be federated (e.g., by datafederation process 1119) from various transmission substation databases1120, primary distribution substation databases 1130, secondarysubstation databases 1140, distribution feeder (or other distributedintelligence point) database 1150. Typically, edge devices (end-points,sites, etc.) need not have further database or storage capabilities, butmay depending upon various factors and considerations of a givenimplementation.

Notably, the architecture herein may build upon the core function groupsconcept above to extend grid capabilities to the control center andenterprise data center levels, using the layer model to unify elementsand approaches that have typically been designed and operated as if theywere separate and unrelated. This model may also be extended to provideservices related to application integration, as well as distributedprocessing. This yields a four tier model, wherein each tier is composedof multiple services layers. The four tiers are as follows (from thebottom of the stack upward), where each of the layers and tiers isintended to build upon those below them:

1. Network services;

2. Distributed Intelligence services;

3. Smart Grid Core Function services; and

4. Application Integration services.

FIGS. 12A-12E illustrates the Layered Services Architecture model(“stack”) 1200. In particular, FIG. 12A shows a full stack model for thelayered services. Application Integration Services 1210 comprisesservices that facilitate the connection of applications to data sourcesand each other. Note that at this top layer the stack splits into twoparallel parts as shown in FIG. 12B: one for enterprise levelintegration 1212 and one for integration at the real-time operationslevel 1214. For the enterprise level, there are many availablesolutions, and the use of enterprise service buses and relatedmiddleware in a Service Oriented Architecture (SOA) environment iscommon. For the real-time operations side, the architecture hereinrelies less on such middleware tools and much more on network services.This is for two reasons: network-based application integration canperform with much lower latencies than middleware methods, and the useof middleware in a control center environment introduces a layer of costand support complexity that is not desirable, given that the nature ofintegration at the real-time operations level does not require the moregeneral file transfer and service composition capabilities of theenterprise SOA environment. The enterprise side of the applicationintegration layer is not actually part of the distributed intelligence(DI) platform; it is shown for completeness and to recognize thatinterface to this form of integration environment may be needed as partof a fully integrated computing platform framework.

Additionally, the Smart Grid Core Function Services layer 1220 (detailedin FIG. 12C) generally comprises the components listed above in FIG. 5,namely services that derive from or are required by the capabilities ofthe smart grid superstructure. Moreover, the Distributed IntelligenceServices layer 1230 (FIG. 12D) comprises support for data processing anddata management over multiple, geographically dispersed, networkedprocessors, some of which are embedded. Lastly, Network Services layer1240 (FIG. 12E) comprises IP-based data transport services for griddevices, processing systems, and applications. Note that CEP isillustratively included here because it is fundamental to networkmanagement in the core grid architecture model.

Another way of approaching the layered services stack as shown in FIGS.12A-12E above is from the perspective of the devices themselves,particularly as a logical stack. For instance, a logical stack 1300 forthe distributed intelligence platform is illustrated in FIG. 13. Notethat not all parts of this stack 1300 are intended to be present inevery processing node in a system. FIG. 13 is correlated with thelayered services stack 1200 of FIGS. 12A-12E, but the logical stack 1300also shows placement of two types of data stores (historian 1365 tostore a time series of data, thus maintaining a collection of (e.g., allof) the past values and database 1336 to store generally only the mostrecent (e.g., periodically refreshed) values of a set of operationalvariables), as well as an API layer 1340 to expose certain capabilitiesof the platform to the applications and to upper levels of the platformstack. Generally, at the base of the stack 1300 is the known IPv4/v6protocol stack 1310, above which are grid protocols 1320 andpeer-to-peer (P2P) messaging protocols 1325. Further up the stack 1300are standard network services 1332, embedded CEP engines 1334, and thedistributed database 1336. Through the API layer 1340, the stack 1300reaches distributed intelligence services 1350 and unifiedcomputing/hypervisor(s) 1355, upon which rest grid-specific networkservices 1360 and historians 1365. Application integrationservices/tools 1370 tops the stack 1300, allowing for one or moreapplications 1380 to communicate with the grid devices, accordingly.

Based on the description above, a layered services platform may becreated, which is a distributed architecture upon which the layeredservices and smart grid applications may run. The distributedapplication architecture makes use of various locations in the grid,such as, e.g., field area network routers and secondary substationrouters, primary substations, control centers and monitoring centers,and enterprise data centers. Note that this architecture can be extendedto edge devices, including devices that are not part of the utilityinfrastructure, such as building and home energy management platforms,electric vehicles and chargers, etc.

FIGS. 14A-14D illustrate an example of the layered services platformdescribed above. For instance, as detailed in FIGS. 14A-14D, enterprisedata centers 1410 may comprise various business intelligence (BI) tools,applications (enterprise resource planning or “ERP,” customerinformation systems or “CIS,” etc.), and repositories based on a unifiedcomputing system (UCS). Other systems, such as meter data managementsystems (MDMS) may also be present. Via a utility tier network 1420, theenterprise data centers 1410 may be in communicative relationship withone or more utility control centers 1430, which comprise head-endcontrol and other systems, in addition to various visualization tools,control interfaces, applications, databases, etc. Illustratively, aservices-ready engine (SRE), application extension platform (AXP), orUCS may structurally organize the utility control centers 1420. Througha system control tier network 1440, one or more primary substations 1450may be reached by the control centers 1430, where a grid connectedrouter (GCR) interconnects various services (apps, databases, etc.)through local device interfaces. Utility FANs (field area networks) 1460(or neighborhood area networks (NAN's)) may then bridge the gap to poletop FARs 1470, or else (e.g., in Europe) secondary distributionsubstations 1480 to reach various prosumer (professional consumer)assets 1490, accordingly.

Layered/Distributed Grid-Specific Network Services

As noted above, conventional grid functions and applications aretypically “siloed”, meaning that each application is independent andspecific to its task; however, many of these siloedapplications/functions have overlapping functionality. For example, afirst application such as volt/VAr regulation may require voltagereadouts, and a second application such as a outage detection may alsorequire the same voltage readouts, which is inefficient.

Accordingly, the techniques described herein treat grid controloperations as layered services that may be distributed in an ‘on demand’or ‘as needed’ basis. There are a finite set of capabilities that, whenintegrated into the network as extended network services, convert thenetwork to a platform for applications. The techniques herein convertcertain siloed functions into layered services and distribute (e.g.,push or pull) these shared services into the network. In other words,while conventional grid functions and applications are typically siloed,which is inefficient as many grid functions/applications haveoverlapping functionality, the techniques herein function to “de-silo”such conventional grid functions and applications and convert them intolayered services amenable to dynamic distribution.

In other words, by implementing a set of capabilities as networkservices, as opposed to applications, the techniques herein create anetwork platform for applications that enables operation of bothdistributed and centralized applications (e.g., power gridapplications). That is, by moving some functions that would previouslyhave been implemented in various siloed applications into networkservices, the techniques herein effectively convert siloed, and oftenrepetitively implemented, portions of grid applications to servicelayers available across an entire distributed platform. In this manner,as described below, the techniques herein make the network moregrid-aware and more intelligent in the context of power grids, such thatapplications development may proceed on a platform that insulates lowerlevel functions from high level application logic, thus simplifyingapplication development and deployment in both centralized anddistributed environments.

Specifically, the techniques herein describe a system that provides aplurality of grid sensors in the utility grid that are configured togenerate grid data values such as, for example, raw grid data values,processed grid data values, and/or any combination thereof, and tocommunicate the grid data values using a utility grid communicationnetwork. The plurality of distributed grid devices in the utility gridare configured to receive the grid data values, and one or more of theplurality of grid devices may be configured to convert raw grid datavalues into processed grid data values. The plurality of applicationdevices in the utility grid are configured to access the grid datavalues from the distributed grid devices, and to further process thegrid data values according to a particular grid application operating atthe corresponding application device into application data values. Inthis manner, data for grid functions/applications may be hierarchicallyprocessed to create multiple grid-specific network layers (e.g., a rawdata layer, a processed data layer, an application data layer, etc.)that can distribute and/or store the data for use by grid functionsand/or applications. Consequently, the techniques herein allow gridapplications and/or functions to access layered distributed data, ratherthan being responsible for autonomously generating their own data (i.e.,unlike a conventional siloed application). For example, if a volt/VArdevice and an outage detection device both require grid datacorresponding to line voltage, rather than having each deviceincorporate its own voltmeter, or otherwise its own application code toacquire line voltage from the same voltmeter, the techniques hereinallow each device to access the line voltage data from an appropriategrid-specific network layer, thereby de-siloing the respective griddevice functions.

Illustratively, the techniques described herein may be performed byhardware, software, and/or firmware, such as in accordance with thegrid-specific application process 348, which may contain computerexecutable instructions executed by the processor 320 to performfunctions relating to the techniques described herein. For example, thetechniques herein may be treated as a “service interface process,” andmay be located across a distributed set of participating devices 300,such as grid sensors, distributed grid devices, grid applicationdevices, and the like, as described herein, with functionality of theprocess 348 specifically tailored to the particular device's role withinthe techniques of the various embodiments detailed below.

Operationally, the techniques described herein allow for alayered/distributed grid-specific network services system,illustratively through the various tiers (i.e., layers) 510, 520, 530,and 540 of smart grid functions stack 500 in FIG. 5. This extends theconcept of network services in a utility grid specific manner.

As an alternative view of the smart grid functions stack 500 in FIG. 5,FIG. 15 illustrates another example smart grid functions stack 1500showing greater detail with respect to the layered/distributedgrid-specific network services techniques specifically described herein.In particular, the basic idea is to start with traditional networkservices, and then add at least three more layers of service sets thatare increasingly smart grid oriented. Similar to layer 510 of stack 500,the bottom layer 1510 is standard network services.

The second layer 1520 provides grid data via remote measurement 1522,which may include, for example, a fabric of grid sensors associated withthe utility grid. For example, each capable node in the network (e.g., agrid sensor in the grid fabric of the utility grid) may collect thelocal “shared grid data” via remote monitoring 1522. Grid data mayinclude raw grid data such as, for example, line voltage data, phasedata, temperature data, metering data, and the like. Additionally, rawgrid data may be subject to low level data processing 1505 to generateprocessed grid data values. As used herein, “grid data” may refer toeither raw grid data or processed grid data. Grid data from second layer1520 may be directly distributed to smart grid applications 1550, orplaced into grid data value storage 1510 (e.g., an appropriatedatabase).

Third layer 1520 provides higher order data processing by mid level dataprocessing 1532, which may include, for example, distributed griddevices that receive grid data (e.g., raw grid data and/or processedgrid data) from second layer 1520 and generate processed grid datavalues that may be directly distributed to smart grid applications 1550,or placed into processed grid data value storage 1534 (e.g., adatabase).

The grid data generated by second layer 1520 and/or the processed griddata generated by third layer 1530 may be subject to further processingby fourth layer 1540. For example, a topology schema (such as, e.g., acommon interface model, CIM) can be applied by high level dataprocessing 1542 to provide a spatial and/or temporal context to thedata. The “shared functionality” may also be dynamically configured. Asan example, in addition to certain standardized grid data values, suchas voltage, phasors, etc., many grid application processes need acomplex event processing (CEP) service to provide particularmeasurements or particular data based on a set or sets of configuredrules. Fourth layer 1540 allows complex grid data that has undergonehigh level data processing 1542 to be dynamically distributed as aservice. For example, fourth layer 1540 may provide CEP as a networkservice, where the necessary CEP rule set, or sets, may be pushed intothe network so that smart grid applications/processes 1550 can make useof them without duplicating the CEP capability in a siloed fashion foreach and every smart grid application 1550. Instead, fourth layer 1540now allows many other applications/processes (or other processes ingeneral) to utilize a shared CEP processing capability along with sharedgrid data. It is contemplated within the scope of the disclosure thateach grid application/process may potentially have its own CEP rule set.Notably, there are a finite set of such capabilities that, whenintegrated into the network as extended network services, convert thenetwork to a platform upon which ecosystem partners may easily placetheir applications.

The techniques herein extract grid data, and various hierarchical levelsof processed grid data, from their typical application/process silos,and convert them into layered services so that they are availableuniversally to all relevant smart grid applications, just as othernetwork services are exposed. As an example, since grid statedetermination arises time and again in distribution grid applications,the techniques herein make grid state determination into a networkservice available to any authorized application. Likewise, controlactuation verification (i.e., determining whether or not a controldevice actually responded to a control command) could be another networkservice. The techniques herein include all steps necessary to providelayerized services for a utility grid: data acquisition, dataprocessing, data conversion into higher order data (e.g., processed griddata values, application data values, and the like), and dynamicdistribution to authorized smart grid applications/process. Thetechniques herein allow for CEP as a service (CEPaaS), grid state as aservice (GSaaS), and the like. Since CEP arises time and again in thesmart grid area, both for sensor data processing, for device eventmanagement, and for network management, CEPaaS can greatly improve theefficiency and scalability of the smart utility grid. It is furthercontemplated within the scope of the disclosure that the various layers(e.g., first layer 1510, second layer 1520, third layer 1520, fourthlayer 1540, and the like) may also function as true distributeddatabases, as in peer-to-peer messaging. The above described serviceshave the benefit of making the network an attractive platform for smartgrid applications. The layered network services element is one part of athree-part structure: networking, embedded/distributed computingplatforms, and finally services that tie the first two together in afashion that simplifies development and implementation of smart gridapplications.

Said differently, according to the techniques herein, to supportdistributed intelligence for electric power grids or any physicalnetwork, the concept of network services may be extended to become astack of service groups, where the services become increasinglydomain-oriented with increasing levels within the stack hierarchy. Inother words, the lower layer contains ordinary network services, thenext layer contains services that support distributed intelligence, thethird layer provides services that support domain specific corefunctions, and the top layer provides services that support applicationintegration for real time systems.

The domain-specific services layer content depends on the nature of thetarget domain. For electric power systems, it may represent a set ofdomain services that support a wide range of common needs forapplications in that particular domain. The concept of layered networkservices is general; however, the nature of the services to be found inthe core functions services layer may be specific to a particulardomain, and so may change depending on that domain. Illustratively, somenon-limiting example services for this layer may include:

-   -   distributed SCADA data collection and aggregation; grid state        determination and promulgation;    -   implementation of distributed analytics for grid data;    -   control command delivery and operational verification;    -   control function federation (merging of multiple        objective/multiple control systems so that common control        elements are used in non-conflicting ways);    -   processing of event streams from grid devices to filter, prevent        flooding, and to detect and classify events for low latency        responses; and    -   providing virtualization of legacy grid devices so that they are        compatible with modern approaches to device operation and        network security.        As discussed above, FIGS. 12A-12E show a full stack model 1200        for the layered services. In particular, as stated above, the        core function groups concept may be built upon to extend grid        capabilities to the control center and enterprise data center        levels, using the layer model to unify elements and approaches        that have typically been designed and operated as if they were        separate and unrelated. As shown above in FIGS. 12A-12E, this        model may be extended to provide services related to application        integration, as well as distributed processing.

In addition, as discussed above, FIGS. 14A-14D above show a distributedarchitecture upon which the layered services and smart grid applicationsmay run, where the distributed application architecture may makes use offield area network routers and secondary substation routers, primarysubstations, control/monitoring centers, enterprise data centers, and soon. As also noted above, this architecture can easily be extended toedge devices, including devices that are not part of the utilityinfrastructure, such as building and home energy management platforms,electric vehicles and chargers, etc.

As also discussed above, FIG. 13 shows a logical stack for thedistributed intelligence platform, where the example stack has a directcorrespondence with the layered network services stack shown in FIGS.12A-12E. To be more explicit, FIG. 13 shows placement of two types ofdata stores 1365 (e.g., a historian, distributed database, and the like)as well as an API layer 1340 to expose certain capabilities of theplatform to the applications 1380 and to upper levels of the platformstack (e.g. application integration services, distributed intelligenceservices, etc.).

To illustrate one or more embodiments of the techniques herein, FIGS.16-18 demonstrate examples of layered/distributed grid-specific networkservices as described above. For instance, as shown in FIG. 16, ageneralized structure of a layered/distributed grid-specific networkservices system may comprise a grid sensor fabric 1610 comprising gridsensors 1612 within a utility grid (e.g., sub-grid 1620, sub-grid 1630,and sub-grid 1640) and/or sensors 1614 peripheral to a utility grid(e.g., a sensor associated with an edge device). Grid sensors 1612 maycommunicate raw grid data to distributed grid device 1616. Generally,distributed grid device 1616 receives raw grid data from grid sensors1612 located within a particular region of the utility grid (e.g.,sub-grid 1620); however, distributed grid device 1616 may also receiveraw data from grid sensors 1612 within more than one sub-grid (notshown), depending on the location of distributed grid device 1616 withrespect to sub-grid configuration. More than one distributed grid device1616 may receive raw data from the same sensor (e.g., sensor 1614).Distributed grid device 1616 may generate processed grid data values,which are communicated to one or more grid application devices 1650 vianetwork interface 1652. Grid application devices 1650 may comprise a bus1660, network interface 1652, power supply 1654, processor 1656, storage1658, and memory 1662, which may further comprise an operating system1664, data structures 1666, and grid application process(es) 1668. Gridapplication devices 1650 are associated with high level processing, andmay generate application data values such as, for example, gridtopology, grid state, and complex event processing (CEP) data.

As shown in FIG. 17, the techniques herein may provide for generation ofordered data that increases in complexity in a manner that parallels autility grid hierarchy (e.g., a smart grid function stack). Sensorfabric 1710 generates grid data (e.g., raw data from a grid sensor 1712or low-level processed data from a smart grid sensor 1715 with memory1716, low level processor 1717, and optionally storage 1718) thatcorresponds to low level data processing stack 1760. Distributed griddevice 1616 comprises a processor 1722, network interface 1724, storage1726, and memory 1728, and may generate processed grid data values thatcorrespond to mid-level data processing stack 1770. Grid applicationdevice 1750 may generate complex data sets such as, for example, complexevent processing (CEP) data that corresponds to high level dataprocessing stack 1780.

FIG. 18 shows a specific and simplified example of the generation ofordered data with increasing complexity. Grid sensor 1812 (e.g., avoltmeter) may generate a raw voltage data value (e.g., “11001010”) thatis communicated to distributed grid device 1820, which processes rawvoltage data value “11001010” into processed data value 1825corresponding to a voltage of 105V. For example, the distributed griddevice 1820 may be configured with raw data conversion tools, such asvarious device calibration databases, in order to determine what the rawdata (11001010) implies from the particular grid sensor 1812. Gridapplication device 1850 may then receive the processed data value 1825(e.g., pushed or pulled from distributed grid device 1820), and mayfurther process the data to generate a grid application data value, suchas an indicia of “decreased voltage.”

Lastly, FIG. 19 illustrates an example simplified procedure 1900 for alayered/distributed grid-specific network services system in accordancewith one or more embodiments described herein, particularly from theperspective of a distributed grid device and an application device. Theprocedure 1900 may start at step 1910, and continues to step 1920,where, as described in greater detail above, grid data values such asraw grid data values (e.g., voltage, temperature, and the like),processed grid data values (e.g., calculated voltage, phasor, and thelike), and/or any combination thereof are generated by a plurality ofgrid sensors in a utility grid configured to generate such grid datavalues. The grid data values of step 1920 are communicated via step 1930using a communication network (e.g., pushed or pulled), and mayaccordingly be parsed to a plurality of distributed grid devices in step1940 that are configured to receive grid data values, where the griddata values may be processed into grid data values (such as, e.g.,sub-system load, etc.). In step 1950, the grid data values of step 1940are received at a plurality of application devices configured to accessthe grid data values from the distributed grid devices, which mayfurther process the grid data values into application data values (suchas, e.g., CEP model, grid state, and the like) by step 1960 according toa particular grid application operating at the corresponding applicationdevice. The application data values may then be dynamically distributedto authorized applications/processes. The procedure 1900 illustrativelyends in step 1970, though notably with the implied ability to return toany of steps 1910-1960 above to further generate, receive, or processgrid data according to the techniques described herein.

It should be noted that while certain steps within procedure 1900 may beoptional as described above, the steps shown in FIG. 19 are merelyexamples for illustration, and certain other steps may be included orexcluded as desired. Further, while a particular order of the steps isshown, this ordering is merely illustrative, and any suitablearrangement of the steps may be utilized without departing from thescope of the embodiments herein.

The techniques described herein, therefore, provide for alayered/distributed grid-specific network services system. Inparticular, the techniques herein provide a grid-specific networkservices system that distributes layered grid data for the use of gridapplications and/or functions. This may increase the efficiency andscalability of grid applications and/or functions by allowing them toutilize common grid data and processing capability localized toparticular grid-specific network service layers. For example, ifmultiple grid applications and/or functions require a particular voltagereadout, that readout can be pushed to the relevant grid applicationsand/or functions that need it from a raw data grid layer, rather thanhaving each grid application or function specifically reach out to avoltage meters/sensors. By incorporating data processing into thelayered network services system, the techniques herein also allow higherlevels of processed data (e.g., complex event processing data) to beincorporated into a layer, and pushed to grid applications and/orfunctions that require the processed data.

Notably, a layered network services architecture approach addressescomplexity management for smart grids at scale, one of the mostchallenging smart grid design issues. Short term adoption of a layerednetwork services architecture allows for efficient transition to newcontrol systems that are hybrids of distributed elements withcentralized management. Later, as smart grid implementations approachfull scale (in any given dimension), complexity management and the othersmart grid architecture issues will benefit from a layered networkservices architecture.

Said differently, now that communications and embedded processingcapabilities are becoming available in forms that utility companies canuse, a major obstacle in the adoption of distributed intelligence isthat utility companies cannot make large changes in their systems indiscrete steps. Rather they must go through transitions that can takeyears to complete. This is due to the nature of their mission and thefinancial realities utility companies face. In practice, utilities needto transition from centralized to distributed intelligence, and tooperate in a complicated hybrid mode for long periods of time, perhapspermanently. This means that the utility service provider needs to beable to roll out distributed intelligence incrementally whilemaintaining full operations over the entire service area, and be able tomodify the distributed architecture appropriately over time andgeography. Simply having a distributed architecture implementation isnot sufficient; it needs to be easily and continually mutable in termsof what functionality is distributed to which processing locations inthe grid and be capable of coexisting with legacy control systems wherethey remain in place.

The present disclosure thus presents one or more specific features of adistributed intelligence platform that supports variable topology overboth time and geography. The platform provides the mechanisms to locate,execute, and re-locate applications and network services onto availablecomputing platforms that may exist in control and operations centers,substations, field network devices, field edge devices, data centers,monitoring centers, customer premises devices, mobile devices, andservers that may be located in power delivery chain entities external tothe Transmission and Distribution utility. These techniques use acommunication network as a future-proofed platform to incrementally andvariably implement distributed intelligence and thereby achieve theassociated benefits without being forced to make an untenable massiveswitchover or to use a single fixed architecture everywhere in itsservice area.

While there have been shown and described illustrative embodiments thatprovide for auto-discovery and provisioning of utility grid controloperation services, it is to be understood that various otheradaptations and modifications may be made within the spirit and scope ofthe embodiments herein. For example, the embodiments have been shown anddescribed herein with relation to electric grids. However, theembodiments in their broader sense are not as limited, and may, in fact,be used with other types of utility grids, such as gas, water, etc., orspecific types of “smart” networks where appropriate. For example, inaddition to utility grids, recent trends indicate that the future willprogress towards sensor-actuator based automation in various sectorsincluding buildings, communities/cities, transportation, energy, etc.Experts predict that in the coming decades there will be a fabric oftrillions of sensor-actuator devices embedded into our surroundings.This fabric will bring about integrated automation that will greatlyimprove the efficiency of the environment/resources as well as thequality of living for the human and living being within the environment.Moreover, while certain protocols are shown, other suitable protocolsmay be used, accordingly.

Illustratively, the techniques herein can span the entire power deliverychain out to and including networks outside of the utility but connectedto it. In addition, the techniques herein apply to all of the otheradjacencies, such as:

-   -   Rail systems—electric rail power control and monitoring, all        rail and car condition monitoring, route control, accident        detection/prevention, mobile WiFi, control centers;    -   Roadways/highways—hazard detection (fog/ice/flooding/earthquake        damage), bridge/overpass structural condition, congestion        monitoring, emergency response support, transit control        facilities;    -   Rivers and canals—locks and dams, flooding detection/extent        measurement, is dikes and levees, flow/depth, traffic flow;    -   Sewage/wastewater/storm drain systems—treatment plants,        flow/blockage monitoring, leak/spill detection;    -   Etc.

The foregoing description has been directed to specific embodiments. Itwill be apparent, however, that other variations and modifications maybe made to the described embodiments, with the attainment of some or allof their advantages. For instance, it is expressly contemplated that thecomponents and/or elements described herein can be implemented assoftware being stored on a tangible (non-transitory) computer-readablemedium (e.g., disks/CDs/RAM/EEPROM/etc.) having program instructionsexecuting on a computer, hardware, firmware, or a combination thereof.Accordingly this description is to be taken only by way of example andnot to otherwise limit the scope of the embodiments herein. Therefore,it is the object of the appended claims to cover all such variations andmodifications as come within the true spirit and scope of theembodiments herein.

1. A system, comprising: a plurality of grid sensors in a utility gridconfigured to generate grid data values selected from the groupconsisting of raw grid data values, processed grid data values, and anycombination thereof, and to communicate the grid data values using acommunication network; a plurality of distributed grid devices in theutility grid configured to receive grid data values, one or more of theplurality of grid devices configured to convert the raw grid data valuesinto processed grid data values; and a plurality of application devicesin the utility grid configured to access the grid data values from thedistributed grid devices and to further process the grid data valuesaccording to a particular grid application operating at thecorresponding application device into application data values.
 2. Thesystem as in claim 1, further comprising: one or more grid controldevices in the utility grid configured to access and analyze theprocessed grid data values and/or the application data values, and todetermine a grid control response.
 3. The system as in claim 1, whereinone or more of the plurality of distributed grid devices are furtherconfigured to store the grid data values.
 4. The system as in claim 1,wherein one or more of the plurality of distributed grid devices arefurther configured to dynamically distribute the grid data values to theplurality of application devices.
 5. The system as in claim 1, whereinone or more of the plurality of application devices further process thegrid data values according to a complex event processing (CEP) model togenerate the application data values.
 6. The system as in claim 5,wherein the application data values represent a state and locality of atleast a portion of the utility grid.
 7. The system as in claim 1,wherein the plurality of application devices are further configured todynamically distribute the processed grid data values and/or theapplication data values to the one or more grid control devices.
 8. Amethod, comprising: receiving, at a particular device of a plurality ofdistributed grid devices in the utility grid, a plurality of grid datavalues selected from the group consisting of raw grid data values,processed grid data values, and any combination thereof; converting theraw grid data values into processed grid data values; and communicatingthe plurality of grid data values to a plurality of application devicesin the utility grid configured to access the grid data values from thedistributed grid devices.
 9. The method as in claim 8, furthercomprising: storing one or more of the plurality of grid data values.10. The method as in claim 8, further comprising: processing the rawgrid data values and/or the processed grid data values into applicationdata values.
 11. The method as in claim 8, further comprising:dynamically distributing the processed grid data values and/or the rawdata values to one or more grid control devices.
 12. A method,comprising: communicating with a plurality of distributed grid devicesin a utility grid by a particular device of a plurality of applicationdevices in the utility grid; receiving a plurality of grid data valuesfrom the distributed grid devices, the grid data values selected fromthe group consisting of raw grid data values, processed grid datavalues, and any combination thereof; and processing the raw grid datavalues and/or processed grid data values according to a particular gridapplication operating at the particular device into application datavalues.
 13. The method as in claim 12, further comprising: communicatingthe plurality of grid data values and/or application data values to oneor more grid control devices in the utility grid configured to accessand analyze the grid data values and/or the application data values, andto determine a grid control response.
 14. The method as in claim 12,further comprising: storing the plurality of grid data values and/orapplication data values.
 15. The method as in claim 12, wherein the rawgrid data values and/or the processed grid data values are processedaccording to a complex event processing (CEP) model to generate theapplication data values.
 16. The method as in claim 12, wherein theapplication data values represent a state and locality of at least aportion of the utility grid.
 17. An apparatus, comprising: one or morenetwork interfaces to communicate with a utility grid computer network;a processor coupled to the network interfaces and adapted to execute oneor more processes; and a memory configured to store a process executableby the processor, the process when executed operable to: receive aplurality of grid data values selected from the group consisting of rawgrid data values, processed grid data values, and any combinationthereof; convert the raw grid data values into processed grid datavalues; and communicate the plurality of grid data values to a pluralityof application devices in the utility grid configured to access the griddata values from the apparatus and other distributed grid devices. 18.The apparatus as in claim 17, wherein the process when executed isfurther operable to: store one or more of the plurality of grid datavalues.
 19. An apparatus, comprising: one or more network interfaces tocommunicate with a utility grid computer network; a processor coupled tothe network interfaces and adapted to execute one or more processes; anda memory configured to store a grid application process executable bythe processor, the process when executed operable to: communicate with aplurality of distributed grid devices in a utility grid; receive aplurality of grid data values from the distributed grid devices, thegrid data values selected from the group consisting of raw grid datavalues, processed grid data values, and any combination thereof; andprocess the raw grid data values and/or processed grid data valuesaccording to the grid application process into application data values.20. The apparatus as in claim 19, wherein the process when executed isfurther operable to: communicate the plurality of grid data valuesand/or application data values to one or more grid control devices inthe utility grid configured to access and analyze the grid data valuesand/or the application data values, and to determine a grid controlresponse.
 21. A system, comprising: a utility grid in which one or morecore grid functions are performed; a computer network configured toprovide communication between network-capable devices of the utilitygrid; one or more network devices within the computer network andconfigured to implement one or more of the core grid functions as one ormore corresponding network services; and one or more network-capablegrid devices within the utility grid and configured to access the one ormore network services from the one or more network devices.
 22. Thesystem as in claim 21, wherein core grid functions are selected from agroup consisting of: measuring line voltage data, calculating phasedata, measuring temperature data, and determining metering data.
 23. Thesystem as in claim 21, wherein network-capable grid devices areconfigured to use core grid functions to produce application data valuesselected from a group consisting of: phasors and complex eventprocessing outputs.